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ABSTRACT 


Traditionally in the Navy/Marine Corps, in an effort to be proactive and prevent 
failures, maintenance and inspections are performed at fixed intervals independent of 
aircraft status. The current preventive maintenance strategy services and replaces certain 
components on a predetermined schedule. Additionally, the current Navy/Marine Corps 
aircraft repair process is reactive. When failures occur, the logistics system — 
maintenance and supply — respond. The Joint Strike Fighter Autonomic Logistics System 
(ALS) is proposed to be better then the logistic system in place. Under the ALS 
maintenance is performed only as needed. The idea is to decrease the logistics 
infrastructure and simultaneously improve logistic performance, by performing 
maintenance only as needed. Additionally, parts are ordered ‘autonomously’ without 
human intervention. The logistics system prepares for an impending failure. In this thesis 
simulations are developed to compare the traditional repair system and the ALS. An 
analysis is conducted to show differences in performance in respect to aircraft 
availability, failures per mission and maintenance-man-hour-per-flight-hour. The ALS 


maintenance model dominated traditional maintenance under the study assumptions. 
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THESIS DISCLAIMER 


The reader is cautioned that the computer program developed in this research may 
not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs and data herein are free of 
computational, logic, and collection errors, they cannot be considered validated. Any 
application of these programs or data without additional verification is at the risk of the 


user. 
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EXECUTIVE SUMMARY 


Traditionally in the Navy/Marine Corps, to proactively prevent failures, 
maintenance and inspections are performed at fixed intervals independent of aircraft 
status. The current preventive maintenance strategy services and replaces certain 
components on a predetermined schedule. Additionally, the current Navy/Marine Corps 
aircraft repair process is reactive. First failures occur, then the logistics system — 
maintenance and supply — respond. 

The Joint Strike Fighter Autonomic Logistics System (ALS) is proposed be better 
then the logistic system in place. Under the ALS, maintenance is performed only as 
needed. The idea is to decrease the logistics infrastructure and simultaneously improve 
logistic performance by performing maintenance only as needed. Additionally, parts are 
ordered “autonomously” without human intervention. The logistics system prepares for 
an impending failure. 

Simulations described in this thesis compare the traditional repair system and the 
ALS. An analysis shows the differences in performance in respect to aircraft availability, 
failures per mission and maintenance-man-hour-per-flight-hour. 

Due to an absence of JSF history and data on JSF maintenance processes, the 
analysis is performed using an existing aircraft. The simulations are based on the Navy’s 
F/A-18E/F jet engine repair process. The aircraft will not be replaced by the JSF, so it 


may be beneficial to alter the aircraft in order to take advantage of the ALS. 


This thesis introduces two simulations. One simulation is for the traditional, 
F414-GE-400 aircraft engine repair process, currently in place. The other simulation is 
for the F414-GE-400 aircraft engine repair process under the ALS. Stochastic models 
developed are used in a face validation of the DES models. Design points for the 
simulations are selected using a Nearly Orthogonal-Latin Hypercube (NOLH) design. 
Regression models are produced to define the relationship between each of the MOEs 
and the predictor variables. The FPM regression models are used to compare the 


traditional repair system and the ALS. 
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The ALS maintenance model dominates in terms of flights per mission (FPM) and 
maintenance man hour per mission (MMHF). However, large gains in operational 
availability were not realized. In terms of FPM, when deciding between investing in 
module reliability in the traditional repair system or prognostic accuracy it is best to 
invest in prognostics and switch to the ALS. In terms of MMHF, the ALS potential far 


exceeds that of the traditional system. The ALS is superior to the traditional system. 
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I. INTRODUCTION 


A. BACKGROUND/AREA OF RESEARCH 

1. Introduction 

In November 2002, Department of Defense (DoD) policy directed the military 
services to implement the tenets of Conditioned Based Maintenance Plus (CBM+) in 
weapons systems and logistics support programs where cost effective. CBM+ focuses on 
predicting maintenance needs and responding accordingly. The idea is to decrease the 
logistics infrastructure and simultaneously improve logistic performance by performing 
maintenance only as needed. “The Joint Strike Fighter (JSF) Prognostics Health 
Management System (PHM) is highlighted as an “emerging” example of a true 
prognostics-capable aircraft embodying the full intent of the CBM+” (Smith, 2003). 


Prognostic capability alone does not improve the entire maintenance and logistics 
system. “Prognostic capability may properly identify a material requirement, but the 
requirement must be transmitted, received, filled, transported, and delivered to provide a 
solution to the maintainer’ (DoD, 2004). <A logistic network where real time 
maintenance and supply needs are known is essential for faster support. This is in 
agreement with the “sense and respond logistics” concept. The customer has only what 
they need when they need it. For this to be viable, “logisticians need to be able to 
communicate with customers, to know where they are geographically, to know what they 


are doing...” (Schrady, 2005). 


To take advantage of its prognostic capability, and to accelerate the logistics 
process, the JSF PHM comes as part of an Autonomic Logistics System (ALS) that 
includes a distributed information system. According to the JSF website: The JSF will 
achieve unprecedented levels of reliability and maintainability, will be the most 
supportable aircraft, and will be ready to fight anytime and anyplace (F-35, 2006). 

Zi Preventive Maintenance/Reactive Support 

Traditionally in the Navy/Marine Corps, to proactively prevent failures, 
maintenance and inspections are performed at fixed intervals independent of aircraft 


status. The current preventive maintenance strategy services and replaces certain 


components on a pre-determined schedule. This periodic maintenance depends strictly on 
flight hours accumulated on the component (age). Preventive maintenance is 
independent of the component’s condition. Scheduled/preventive maintenance wastes 
time and money; some of the scheduled maintenance is unnecessary. Only 23% of 
aircraft equipment failure patterns are age related, 77% are random (Sondalini, 2003). 


For random failures, one cannot predict when a failure occurs based only on age. 


Additionally, current logistics and maintenance support are reactive. First failures 
occur, then the logistics system — maintenance and supply — respond. The maintenance 
personnel troubleshoot to isolate the problem, then order the part and wait for supply to 
requisition the request. Time is wasted while waiting on parts. The Automated Logistics 
system (ALS) remedies these two areas: scheduled maintenance based on age and 
responsiveness to failure. 

3. Autonomic Logistics System 

An Autonomic Logistics System (ALS) is a system that automatically responds to 
maintenance/failure events. Tasks such as identifying failures and ordering parts are 
executed without human intervention. “The JSF Autonomic logistics system 
encompasses three essential components: (1) a highly reliable, maintainable, and 
intelligent aircraft that incorporates Prognostics Health Management (PHM) technology; 
(2) a technologically-enabled maintainer, capable of effectively and efficiently 
maintaining the JSF; and (3) a Joint Distributed Information System (JDIS) that 
incorporates advanced information system technology to provide decision support tools 
and an effective communication network linking the JSF with the logistics infrastructure” 


(Hess, 2004). 


The key to the JSF Autonomic Logistics is the ability to detect and predict 
failures. These capabilities in conjunction with the other essential components not only 
have the potential of saving time but make a policy of condition-based maintenance a 
realistic one. This way maintenance is performed only as needed. Another benefit to 


predicting failures is to stop disabling failures and fatal failures before they occur. 


The PHM technology incorporates: 


e Diagnostics — the process of determining the state of a component to 
perform its function(s) 


e Prognostics — predictive diagnostics which includes determining the 
remaining life or time span of proper operation of a component 


e Health Management — the capability to make appropriate decisions about 
maintenance actions based on _ diagnostics/prognostics information, 
available resources and operational demand” (Hess, 2006). 


Ideally the PHM detects/predicts a component failure and instantaneously relays 
this immediately to the appropriate entities. The ALS automatically provides details of 
the cause and location of the failure; troubleshooting is eliminated. Additionally, 
required parts are ordered automatically eliminating the requirement to manually order 
parts and reducing the time for a part to arrive. The ALS will provide the required 
number of personnel to repair the failure and will locate available parts. Knowing a 
failure will occur in advance allows additional time for preparation and flexibility in 
scheduling maintenance. For example, if a failure was detected 30 hours in advance, the 
squadron may decide to continue using the aircraft until the parts arrived. See Figure 1 


for overview of process. 
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Figure 1. JSF Autonomic Logistic Flow Chart 


4. Assumed Benefits of the ALS 

By predicting failures and accelerating information flow the ALS has potential to 
decrease the downtime of the aircraft and to reduce cost. A financial advantage may exist 
despite the fact there is an expense in setting up and maintaining the ALS. An 
Army/Marine CBM-+ initiated program to develop prognostics on helicopters has proven 
to increase readiness. However, few helicopters are installed with the equipment, as costs 


exceed $1 million dollars per aircraft (Messenger, 2004). 


The expectations for the ALS are high. It is anticipated there will be a forty to 
fifty percent savings in manpower and infrastructure cost at unit level while 
simultaneously turn around time will be in minutes instead of hours (Adams, 2003). 
Supposedly, aircraft readiness will increase, down time for repairs will decrease, required 


number of maintenance personnel will decrease, and reducing cost. 


Dr. Scheuren heads the Defense Advanced Research Projects Agency’s (DAPRA) 
Joint Advanced Strike Technologies (JAST) Program. In his view, “both the ultimate 
objective of PHM and its related autonomic logistics system is to reduce maintenance 
manpower requirements by approximately 20% to 40%, increase combat sorties by 25%, 
and reduce the complexity of the logistics trail by 50%, compared to current military 
strike aircraft; all at a cheaper life cycle cost as compared with legacy aircraft” 


(Nickerson, 1998). 


Rebulanan constructed a simulation of the basic framework of an ALS (ALSim) 
as a tool to allow comparison between ALS and the current maintenance process 
(Rebulanan, 2000). His model showed that higher aircraft availability could be obtained 
with an ALS. Rebulanan assumed the prognostics were 100% accurate and maintainers 
were always available. One would expect ALS with 100% accurate prognostics to 
perform better than the current system. What if the prognostics were not 100%? In this 
case, false detections and/or missed detections are possible. With a false detection, the 
logistics system is unnecessarily burdened. Advantages and disadvantages for the current 


and ALS processes are summarized below. 


Current System: 


Advantages 


Scheduled/Preventive maintenance may prevent catastrophic failure, thus 
exposing the underlying cause of such a failure. Under ALS these causes 
may be masked. 


Disadvantages 


Fault occurs, then maintenance must wait for aircraft to land to determine 
problem. 


High time to troubleshoot aircraft (includes diagnostics, it takes time to 
retrieve information) 


Time taken to order part and wait for delivery (aircraft may be unusable). 


Wasted resources: scheduled maintenance replaces parts that may not need 
replacing. Other maintenance performed that may not be required. 


Wasted time: unnecessary scheduled maintenance takes time. 


ALS system: 


Advantages 


Fault detected before it occurs, and information is relayed instantaneously. 
Parts are automatically ordered, saving time. 

No troubleshooting; therefore saves time. 

Can plan maintenance for aircraft at ideal time, 1.e. when part is delivered. 


No scheduled/preventive maintenance saving man-hours, cost and 
increasing readiness. 


Disadvantages 


False alarms may cause unnecessary part replacement and maintenance. 
Chance of not detecting a failure. 


The time from detection of a degraded component until failure may be too 
short or too variable to gain benefits. 


Cost of sensors, communication equipment, etc. 


B. RESEARCH OBJECTIVES / PURPOSE 


The JSF ALS promises to perform better than compared to the current system. 


The intent of the thesis is to examine how the ALS performs. The ALS automatically 


orders parts based on the PHM. For a PHM with various degrees of accuracy, failure 


detection times and false alarm rates - including JSF ORD (operational requirement 
document) requirements - determine the logistic implications, e.g. parts ordered in error 
that were not needed. Indicators of how well the logistic system is doing include: time for 
maintenance to receive a part, time aircraft is down and number of components replaced 
needlessly. Additionally, this thesis estimates the maintenance time saved compared to 


using current policies and the difference in aircraft readiness. 


Critical factors for the traditional repair system and the ALS are identified. The 
factors are used to compare both systems. The purpose of this thesis is to provide 
analytic support for the hypothetical benefits of the ALS over the current system. The 
model will aid in determining how the ALS will perform, and therefore, help decision 


makers determine how much to invest. 


C. SCOPE AND ASSUMPTIONS 

The current logistics structure with the ALS is shown in Figure 2. The JDIS 
component is assumed to work perfectly, and so will not be considered. If the PHM on 
board predicts a failure, a signal is generated. Signals from the aircraft are assumed 
instantaneously received, without interference. Information is available real time. The 


signal initiates what parts are ordered and the maintenance actions needed. 


The ALS still requires human involvement to accomplish tasks. As an example 
parts are ordered autonomously but a supply clerk is needed to fill the requisition. 
Therefore, there may be delay for a human to respond to the message. The autonomic 
driven actions respond immediately to the signal. Human response to the signal depends 
on availability and capability of personnel. However, in this study, it is assumed that 


capable personnel are always available. 


Prognostics are not presumed to be 100% correct/accurate. As a result one of the 
following scenarios may occur: 


(1) PHM accurately predicted the failure. 


(2) False alarm: PHM sends signal when no failure is to occur. 


(3) PHM does not predict/send signal before the failure actually occurs. 
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Additional Assumptions: 
(1) Diagnostics are 100 % accurate. Once a failure is predicted, the fault is 


accurately isolated. 


(2) The manufacturer/depot has unlimited resources and replaces all non- 


ready for issue (NRFI) line replaceable units (LRU) immediately. 


Signal sent from aircraft (A/C) ——~> 
Path of RFI (ready for issue) pat ===> 
Path of NRFI (Not ready for issue) part —> 


Squadron (SQUAD) where maintenance performed 


Figure 2. JSF Autonomic Logistic Structure 


D. PRIOR WORK 

In 2000, Capt. Rene Rebulanan, United States Air force, wrote a thesis to examine 
how the ALS performs and what demands it places on the logistics infrastructure 
(Rebulanan 2000). Using the Java programming language he developed an ALS 
simulation model. For a set of four aircraft he showed the difference in flying the aircraft 
with ALS and without ALS (the traditional system). Aircraft characteristics were based 
on the F-16. The simulation was run for to simulate six months. Rebulanan concluded 


that the ALS leads to higher availability. Rebulanan assumed the prognostics worked all 


the time, that is impending failures were always detected. This thesis explores the impact 
of prognostics that are not 100% accurate. Additionally, this thesis implements a 
squadron worth of aircraft, and analyses the impact on the O and I levels over the lifetime 


of the aircraft (assumed to be 25 years). 


In 2003, Lieutenant Commander, Eric J. Schoch, wrote a thesis that models the 
F414-GE-400 engine repair process. The goal of the simulation was to provide 
operational availability and probability to spare the process given the current system in 
place, which does not incorporate the ALS. His simulation utilizes the Java package 
Simkit, a software package for implementing Discrete Event Simulation (DES) models 
(Buss, 2001). Schoch’s model accounts for all the squadrons and AIMDs in the United 
States Navy. This thesis will implement only one AIMD and the squadrons it supports. 
Schoch’s model mimics the depot with same level of detail as the AIMD, and depot 
inventory levels and turn around times are considered. The logistic impact on the depot 
falls outside the scope of the thesis. The depot is assumed to have a replacement on hand. 
This thesis develops a simulation for the F414-GE-400 repair process augmented with 
ALS. 


E. METHODOLOGY 

Because of the absence of history and data on the JSF maintenance processes, the 
analysis will be performed using an existing aircraft. The model is based on the Navy’s 
F/A-18E/F jet engine repair process. Therefore, model runs are based on the Naval 
Aviation Maintenance Program (NAMP). Two F-414-GE-400 engines power the F/A- 
18E/F. The line replacement units (LRU) simulated are the F-414-GE-400 engine and its 


subcomponents. 


A Discrete Event Simulation (DES) model represents the current F-414-GE-400 
engine repair process. With a DES model one can observe the behavior of the system 
over time. The DES program is based on Lieutenant Commander Schoch’s simulation 
work so it is also implemented in Simkit (Schoch, 2003). Multiple runs (with multiple 
replications) for the current system model are used. The effect of changing the input 


parameters on the measures of effectiveness (MOE) is explored. 


The model will then be augmented with the essential elements of the ALS: PHM 
and JDIS. The ALS model is run multiple times using various prognostic accuracies, 
detection times and false alarm rates. Example ranged include prognostic accuracies: 90, 
95 and 100%, detection times: 10, 20 and 30 hours before failure time and mean flight 
hours between false alarms 450 and 700. The same LRU failure patterns for both the 
legacy system and the ALS are used. For each level of accuracy, expected different 
values of the MOE are observed. For some level of accuracy and below the ALS may 


offer little or no improvement. 


Simulation results for each system are compared to determine the benefits of the 
ALS. Three MOEs are considered. The primary MOE is operational availability: time the 
aircraft are available for a mission divided by the total time of the simulation run. The 
other MOEs considered are maintenance man hours per flight hour (MMHEF) and number 


of failures per flight mission (FPM). 


Additionally, a stochastic model is developed for each system. The purpose of 
the stochastic model is to analyze key elements of the logistics system. The results from 


the stochastic model are a face-validation of the DES model. 


The thesis consists of five steps: 


1. Create a Discrete Event Simulation (DES) using Simkit for the current 
F414-GE-400 engine repair process. 


Augment the current F414-GE-400 engine repair process model with ALS. 
Develop stochastic models and use them to verify the DES model. 
Exercise the DES model using F/A-18E/F data. 


i es GS 


Analyze results of simulation and draw conclusions. 


The analysis shows that ALS maintenance model dominates in terms of flights per 
mission (FPM) and maintenance man hour per mission (MMHF). However, large gains in 
operational availability were not realized. In terms of FPM, when deciding between 
investing in module reliability in the traditional repair system or prognostic accuracy it is 
best to invest in prognostics and switch to the ALS. In terms of MMHF, the ALS 
potential far exceeds that of the traditional system. Thus, the analysis supports the 
conclusion that ALS is superior to the traditional system. 
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F. ORGANIZATION 

Chapter I, Logistics Process, describes the current F414-GE-400 aircraft engine 
repair and the F414-GE-400 engine repair with ALS processes. Additionally Chapter II 
covers data used to model the engine repair process. Chapter III describes the DES 
model. Chapter [IV contains the stochastic models. In this chapter, results from 
simulations using the DES and stochastic models are compared. Chapter V discusses the 
experimental design. Chapter VI is the analysis of the MOEs. Finally, Chapter VII 


provides conclusions and further recommendations. 
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Il. F414-GE-400 AIRCRAFT ENGINE REPAIR PROCESS 


Due to the absence of history and data on the JSF engine maintenance processes, 
the F/A-18E/F ‘Super Hornet’ engine, F-414-GE-400, is used for this study. The F/A - 
18E/F is the latest tactical aircraft to enter the United States Navy. It will not be replaced 
by the JSF so it may be beneficial to alter the aircraft in order to take advantage of the 
ALS. The purpose of this chapter is to describe the F414-GE-400 Aircraft Engine Repair 
Process and give an overview of the data and assumptions. 

A. F-414-GE-400 ENGINE 

Two engines power the F/A-18E/F. The F-414-GE-400 consists primarily of six 
modules: The fan, compressor, combustor, high pressure turbine, low pressure turbine 
and afterburner. The engine is designed for easier more efficient maintenance and the 
modules are fully interchangeable. 


Combustor 






Compressor 


ex V) | vad : 
a B = Afterburner 
High Pressure Low Pressure 
Turbine Turbine 


Figure 3. F414-GE-400 (General Electric, 2003) 


B. THREE LEVELS OF MAINTENANCE 

The Naval Aviation Maintenance Program (NAMP) is founded upon the three- 
level maintenance concept: Organizational (O), Intermediate (I) and Depot (D). As the 
operating unit, the squadron performs the lowest level of maintenance: organizational 


13 


maintenance. The Aviation Intermediate Maintenance Depot (AIMD), located at air 
stations, as well as deployed aircraft carriers, performs the intermediate level 
maintenance. The naval aviation industrial establishment performs the depot level 
maintenance. For the FA-18E/F, the engine repair process is identical while afloat or 


onshore. 


The squadron performs maintenance to support its daily operations. The squadron 
performs limited repair, which includes engine trouble shooting, and removal and 
installation. The engine is removed when a failure that cannot be repaired occurs or 


when one its modules is scheduled for replacement. 


Traditionally, AIMD has extensive maintenance capabilities. However, I-level 
support for the F14-GE-400 is limited. Limited I-level support is also planned for the 
JSF. The AIMD has the ability exchange modules on an engine and assemble engines. 
The AIMD does not repair modules. Modules in need of repair or preventive 


maintenance are sent to a naval aviation industrial establishment. 


For the F414-GE-400, there is only one naval aviation industrial establishment, 
the Naval Aviation Depot at Jacksonville, Florida. “The D-level is the top echelon of the 
jet aircraft engine repair process and can perform all maintenance and repair action” 


(Schoch, 2003). 


Cc. F/A18-E/F ENGINE REPAIR CYCLE 

When an engine failure is detected, the squadron troubleshoots to isolate the cause 
of failure. The engine is removed if the failure cannot be repaired at the O-level. 
Additionally, engines are removed if any of the modules has reached ‘high time.’ High 
time is a predetermined time when the module is scheduled for removal. High times 
prescribe the removal of modules based on hours of operation. When a module reaches 
high time, it is removed regardless of its current condition. Module high times are listed 


in Appendix A. Figure 4 illustrates the O-level echelon maintenance. 
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Engine Failure — 
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Test Aircraft — 


B ir? i i i 
O-level Repair? Minor failure repair Aircraft Up 


Module reaches 


High Time Remove Engine Install Engine 


Engine to AIMD for Ready for Issue 
repair Engine from AIMD 


Figure 4. O-level F414-GE-400 Engine Repair 


Managing an engine pool and a module pool is a key function of the AIMD. The 
AIMD has the ability to disassemble and assemble engines. Modules that need repair, to 
include preventive maintenance, are sent to the depot. A squadron that needs an engine 
receives a ready for issue (RFI) engine from the pool. As long as there is an engine in the 


inventory, the squadron does not have to wait for the removed engine to be repaired. 
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AIMD inspects all engines received from the squadrons. Modules in need of 
repair or preventive maintenance are replaced. AIMD has the option of replacing 
modules with modules in the pool or usable modules from other engines waiting 


assembly. As engines are assembled, they are added to the pool. 


At the I-level, a module is considered in need of preventive maintenance if it has 
reached high time or if it is within a certain range of high time. This range of time is 
called ‘build window.’ For example, if an engine with a build window of twenty hours is 
sent to the AIMD, modules within twenty hours of high time are replaced. Figure 5 


illustrates the I-level echelon maintenance. 
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Figure 5. I-level F414-GE-400 Engine Repair 


The Naval Aviation Depot at Jacksonville, Florida repairs modules. 


AIMD, the depot has an inventory pool in which to issue modules. 


ALS F/A18-E/F ENGINE REPAIR CYCLE 


It is the 


responsibility of the Depot to replace modules turned in from the AIMD. Like the 


Under the ALS, aircraft are enabled with the PHM system. The system monitors 
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the current condition of the engine and forecasts impending failures. In flight, when the 


PHM detects any degradation of the engine, it will isolate the failure and predict the 
remaining life of the engine. Note that the PHM is not 100% accurate. False detections 
and/or missed detections may occur. A signal containing this information is sent from 
the aircraft to each of the three levels of maintenance thereby autonomously initiating 


action at each level. 


The squadron does not troubleshoot engines nor does it remove modules on the 
high time schedule. Upon receiving an impending failure message the squadron will 
decide when to remove the engine from the aircraft. Ideally, the engine removal can 
occur to coincide with the arrival of the replacement engine. Aircraft will no longer sit 
inoperable while waiting for a replacement. The longer the lead time (predicted 
remaining life of engine) the more flexibility the squadron has. The squadron has two 
options: to abort the mission then ground the aircraft, or not ground the aircraft and 
schedule the aircraft for follow on missions. Figure 6 illustrates the O-level echelon 


maintenance at the squadron. 


Test Aircraft — 
Aircraft Up 


Signal from aircraft 


Remove Engine Install Engine 


Ready for Issue 
engine from AIMD 


Engine to AIMD 





Figure 6. ALS O-level F414-GE-400 Engine Repair 


When an impending failure message from the aircraft, is received at the I-level, an 


engine is sent to the squadron. The degraded engine is brought back to the AIMD. The 
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AIMD no longer performs inspections. The PHM directs replacement of the module that 
caused the failure. The degraded module is then sent to the depot. The depot 
automatically sends a RFI module to the AIMD and waits for the degraded module. 
Figure 7 illustrates the I-level echelon maintenance. 
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Remove indicated 
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Depot for repair engine 


Module Pool 


Module from 
Depot 


Figure 7. ALS I-level F414-GE-400 Engine Repair 
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2003): 


ASSUMPTIONS 


1. 


Assumptions Applicable from Schoch’s Thesis 


The simulation uses the following assumptions from Schoch’s thesis (Schoch, 


2. 


All levels of repair operate 24 hours a day. 

A year, for the purpose of this simulation, is exactly 52 weeks long. 
Engines and modules are always repaired and never condemned. 
The D-Level repairs all modules, the I- Level does not. 

The I-Level assembles all engines, the D- Level does not. 


Engine repair at the I-Level begins as soon as there are enough modules to 
complete an engine. 


The O-Level has unlimited engine removal and installation capacity. 
The shipping times between the I and D-Levels are a constant. 


The O-Level removes an engine from a F/A-18 only for module failures 
and high times. All other maintenance requirements have a negligible 
effect on overall operational availability. 


F414-GE-400 engine failures are the result of independent failures of the 
modules in the engine. 


Once an F/A-18’s flight schedule is set, the F/A-18 does not vary from it 
(except in the case of engine failure or high time because it is not flyable). 


If a F/A-18 experiences an engine failure, it lands immediately. No 
emergency diverts, aircraft lands back at base. The model assumes no 
return to origin delay. 


A F/A-18 may have both engines fail simultaneously. 


Additional Assumptions 


The simulation uses the following additional assumptions: 


The depot is assumed to have unlimited resources and therefore always 
has modules on hand ready for issue. 


The intricacies of the depot (e.g. man-hours) are not analyzed. 
There has been a recent push to privatize the Depot. 


Lockheed Martin will provide depot level support for the JSF under a 
Performance-Based Logistics (PBL) agreement. 


Because, a PBL agreement is to guarantee part delivery within a certain 
time, it is assumed assigning various values to the shipping time between 
the depot and the AIMDs captures the impact of depot level support. 
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e Upon repair, modules are good as new. 


° For the ALS, an F/A-18’s flight schedule is set, the F/A-18 does not vary 
from it (except in the case of detected pending engine failure or engine 
failure because it is not flyable). 


e Once an RFI engine is installed in an aircraft, it is considered ready for 
normal flight. Functional check flights are not required. 


° Cannibalization, taking parts off one aircraft to install in another is not an 
option. 
e All aircraft enter service at the same time. 


° All flights scheduled for 2.75 hours. 


F. DATA 

1. Data Source 

The following data/values were obtained from Lieutenant Commander Schoch’s 
(Schoch, 2003) thesis: module failure data, module high times, O and I level module 
engine removal, installation and inspection times, time to transfer engine from AIMD to 
squadron, number of squadrons and aircraft, and engine, module AIMD allowance. All 
values, except the module failure data, are listed in Appendix A. 

2. Data Analysis of Module Time Between Failures 

Text files containing time between failures data for each module type were 
analyzed. Using JMP statistical software, the Exponential, Weibull, Gamma, Normal, 
LogNormal and Beta distributions were fit to each module failure data. The Beta 
distribution is the best fit for each module type. The goodness-of-fit test yielded the 
following p-values: 

e Module 1 (Fan) = 0.2500 

e Module 2 (Compressor) = 0.2500 

e Module 3 (Combustor) = 0.1820 

e Module 4 (HPT) = 0.2500 

e Module 5 (LPT) = 0.2500 

e Module 6 (After Burner) = 0.2500 

For each module type, evidence against the null hypothesis: Data is Beta 


distributed, is not significant. 
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G. CONCLUSION 

The preceding section described the current, traditional logistic system which 
facilitates the F-414-GE-400 engine repair process. Then the F-414-GE-400 repair 
process was described under the ALS. The key components necessary to model the 
system are identified. The traditional logistic system and the ALS are different. The 


components and assumptions mentioned in this chapter are used to model the systems. 
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Ul. DISCRETE EVENT SIMULATION (DES) MODEL 


Simkit event graphs aid in describing the simulation models for both the 
traditional and ALS systems. Event graphs also highlight the difference between the two 


models. 


A. DES, EVENT GRAPHS AND SIMKIT 

1. DES Paradigm 

DES is a modeling paradigm in which the model’s state remains constant except 
for particular events, which can take place at any place or time (Law and Kelton, 2000). 
Events are state transitions. The state of a system is the collection of variables necessary 
to describe the status of the system at any given time (Winston, 2004). The state changes 
at discrete points in time. Scheduled events are placed in a Future Event List (FEL) in 
chronological order. During the simulation, time is advanced in discrete steps to the next 
earliest event in the FEL. 

2. Event Graphs 

Event graphs are diagrams of physically representing discrete event simulation 
models (Buss, 2002). Events are represented by nodes. Directed edges are the 
scheduling relationship between pair of nodes. Nodes are the state transitions and arcs 
perform the scheduling. For the diagram below, if (i == true) then the occurrence of A 


causes the scheduling of B to occur after a delay of time t. 


Figure 8. Event Graph Notation (From: Buss,2001,2) 
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3. Listener Pattern 

A useful application in DES is the ability to make simulated components “listen” 
to another’s events. When a scheduled event of the source component occurs, the event 
is simultaneously scheduled for all components listening to the source with the same 
event. In Figure 9 below, class B listens to class A. The arrival method in class B is 
scheduled by the self-scheduling arrival method of class A. For more information on DES 
and Simkit, see the reference (Buss, 2002). 


CLASS A CLASS B 


Arrival Process 


Arrival Process 





Figure 9. Class B ‘Listens’ to Class A 


4. SimKit 

All models in this chapter have been developed using Simkit. Simkit is a DES 
implementation software package written in Java, which directly supports building event 
graph models. Every event in an event graph model corresponds to an instance method in 
a Simkit class. The relationship between Simkit and event graph modeling is summarize 
in table 1. More information on how to use Simkit to create DES models can be found in 
(Buss, 2002). Simkit can be downloaded from the internet at the following URL: 
http://diana.or.nps.navy.mil/simkit/ 
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Event Graph Simkit 

















Parameter Private instance variable, setter and getter 

State Variable Protected instance variable getter, no setter 

Event ‘do’ method 

Scheduling Event Call to waitDelay() in scheduling event's ‘do’ method 
Run Event Reset() method to initialize state variables; 


doRun() method to fire Property Change events for time- 
varying state variables 











Events scheduled from Run event | Call to waitDelay() in doRun() method 





Table 1. —_ Relationship between Simkit and Event Graph modeling 


B. DES MODEL FOR TRADITIONAL ENGINE REPAIR SYSTEM 

As mentioned in chapter one, this thesis builds on Schochs’ thesis work. His 
model simulates the current/traditional F414-GE-400 engine repair process. The three 
levels of maintenance, O-level, I-level and D-level are represented. Simulates F/A-E/F 
flight schedules and engine failures are used to populate the engine process. The twelve 
Java classes of Schoch’s DES model follow with a brief description (Schoch, 2003). 

1. Module Type Class 

This class sets up the types of modules that can be used in a simulation. For the 
F414-GE-400 engine repair process simulation there are six types of modules. 

2. Module Class 

An object of this class represents a module. The module can be of any type 
specified by the module type class, for example: fan. A failure time is randomly selected 
from a data set corresponding to the module type. This object tracks component hours 
used and knows when it has reached high time or time of failure. 

3 Engine Blueprint Class 

This class details the specifications each engine must meet: position and type of 


each module, build window and high time schedule. O-Level engine removal, 


Ze 





installation and troubleshooting times are set in this class. Additionally, I-Level engine 
inspection time and module removal/installation time for each module type are set in this 
class. 

4. Engine Class 

An object of this class represents a F-414-GE-400 engine. It is made up of one of 
each type of module, for a total of six modules. It tracks engine hours and screens each 
module if it has reached high time or time of failure. The time to high time corresponds 
to the module time to high time with lowest value. The time to failure corresponds to the 
module failure with the lowest value. Each engine object must conform to the 
specifications of the Engine Blueprint class. 

5. F18 Hornet Class 

An object of this class represents an F/A-18E/F aircraft. The F/A-18E/F is 
simulated by using two engine objects and one FlightSchedule object. The aircraft object 
tracks its own hours. Time to high time corresponds to the engine time to high time with 
the lowest value. Time to failure corresponds to the engine time to failure with the 
lowest value. When an aircraft reaches time of failure of high time it is no longer flyable. 

6. Flight Schedule Class 

This class provides a selection of schedules an F/A-18E/F Hornet object. The 
schedules are arrays. Elements indexed by odd numbers in the array are ground times 
(time between flights). The even indexed elements in the array are flight times. The final 
odd placed element in the array is a zero. This element directs the F/A-18E/F Hornet to 
begin the flight schedule again. 

Te O-level Class 

An object of this class represents a squadron. When a supported aircraft reaches 
high time or has an engine failure, the squadron removes the engine. The squadron 
requests a replacement engine from its supporting I-level. Upon arrival, the replacement 
engine is installed in the aircraft by the O-level object. 

8. I-level Class 

An object of this class represents an AIMD. It delivers RFI engines to the O- 


levels it supports. A linked list keeps track of all good modules assembled on engines 
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awaiting part and in inventory. The best engines are built by screening and using 
modules with the largest amount of time remaining before high time is reached. 

9. D-level Class 

An object of the class represents a depot. Like an AIMD, the depot has linked 
lists which keep a track of all its modules. A module received from the AIMD is placed 
in the needs repair linked list. 

10. F18 Simulation Manager Class 

The entire infrastructure, setup in the F18 Simulation Setup class is created here. 
It determine out how many modules and engines are needed based on the number of 
aircraft and inventories at all I and D levels. This class keeps a record of every object 
made in the simulation and therefore serves as the report generator for the simulation. 

11. F118 Simulation Randomness Class 

All randomness in the simulation is generated in this class. Module times between 
failures are randomly selected, with replacement, from data that contains actual failure 
times. 

12. F18 Simulation Setup Class 

This class controls the entire simulation. This class is used to construct the 
infrastructure to be modeled in the simulation. The number of aircraft, squadrons, 
AIMDs, and I-level and D-level inventories are set up in this class. An entity knows 
what it supports because, using Simkit, it is registered to listen to it. Figure 10 shows the 


listening relationship between the object of each type class. 


O Level objects listens to Hornets ILevel objects listens to OLevels DLevel objects listens to [Level 
F18Hornet objects OLevel objects ILevel Objects DLevel Objects 









































Hornet object listens to supporting OLevel  OLevel object listens to supporting [Level [Level object listens to supporting DLevel 


Figure 10. Listening Relationship of Simulated Objects 
| 


C; CHANGES TO SCHOCH’S MODEL 

Two logistics systems are modeled: traditional and ALS. Schoch’s simulation, 
with some adjustments, is used to model the traditional system. His simulation is altered 
for the ALS model. Differences for the traditional model include: 


e For failure times, Schoch randomly selected failure times from data sets. 
In this thesis, a probability distribution is fit to each data set. These 
distributions are used to generate the failure times. 


e Schoch modeled the depot in detail. This thesis has less depot detail. In 
this thesis, the depot has unlimited resources. Depot inventory levels and 
maintenance man hours are not considered in this thesis. Impact of depot 
level support is captured by assigning various values to the shipping time 
between the depot and AIMDs. 


e Variables are added to the Schoch model to track squadron and AIMD 
man hours, number of requisitions made, number of requisition filled, 
number of requisitions gone off base, number of in-flight failures, and 
number of missions completed. 


In addition to the differences above, for the ALS system changes to the Schoch 
model include: 

e No high times. 

e Prognostics (the ability to detect pending failures). 


D. DES MODEL FOR ALS ENGINE REPAIR PROCESS 

The model in this thesis simulates the F-414-GE-400 engine repair process in 
accordance with the ALS. The O-level and I-level are modeled with detail. The 
simulated F/A-18E/F aircraft are enhanced with prognostics. If an impending failure is 
detected, the aircraft send the information to each of the three levels of maintenance 
thereby initiating action at each level. 

1 Be Engine Blueprint Class 

As in the traditional model, all engines in the ALS must conform to the 
specifications detailed in the engine blueprint class. Prognostic accuracy and detection 
time before failure (lead time -remaining life after detection before the failure occurs) are 


set in this class. 
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2. F18 Hornet Class 

The aircraft object keeps a track of its own flight hours. When the prognostics 
detect an impending failure, the O-level and I-level are instantly notified. Autonomously, 
a replacement engine is delivered to the squadron. The aircraft keeps making its 
scheduled flights until one of the following occur: the prognostic works properly and a 
replacement engine arrives, the prognostics works properly and the impending failure is 
predicted to occur during the next flight, or the prognostics do not work and an in flight 
failure occurs. The engine detection time check method screens for these events. The first 
two events result in the grounding of the aircraft. If the prognostics do not work and a 
failure occurs, then the ALS system responds in the same way as the traditional (Schoch) 
system. Additionally, a false positive time is generated for each aircraft object. When the 
aircraft reaches the false positive time, a module is randomly chosen for the prognostics 
detect an impending failure. The ALS F-18 Hornet event graph is displayed Appendix B. 

3. O-Level Class 

The O-level class keeps track of engine requests made from all the aircraft it 
supports. Additionally, it tracks maintenance hours. If the engine to be replaced has 
failed, the squadron responds in the traditional manner and begins troubleshooting. If the 
request was made by the ALS, the squadron does not troubleshoot the engine. The 
squadron waits until the replacement engine arrives then replaces the engine. The ALS O- 
level event graph is displayed in Appendix C. 

4. I-Level Class 

The I-level class keeps track of engine requests made from all the squadrons it 
supports. Additionally, it tracks the number of RFI modules ordered from and received 


from the depot. It also tracks its maintenance hours. 


Engines received from the squadron have either failed or have an impending 
failure that has been detected. If an engine is received due to the prognostics, 


troubleshooting is not required. If a failed engine is received, troubleshooting is required. 


Modules that have failed or those with detected impending failures are removed 
from the engine and sent to the depot for repair. In the case where module inventory is 


empty, additional modules may be removed from an engine in order to complete the 
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assembly of another engine. This practice is called cannibalizing. Cannibalizing is an 
allowed practice at the I-level. The ALS I-level Event graph is displayed in Appendix D. 

3. D-level Class 

The depot always has a replacement module always on hand. The level of service 
of the depot is captured by altering the I-level to D-level shipping times. The D-Level 
class keeps a track of the number of requisitions made to the depot. The D-Level Event 
graph is displayed in Appendix E. 

6. Stochastic Class 

Failures depend on module type. The probability distributions for module failures 
are set in this class. Additionally, the distribution that generates the false positive signals 


is set here. 


E. CONCLUSION 

DES models have been developed to simulate the traditional logistic system and 
the ALS. The characteristics and behavior of the system components are captured in the 
DES models. Java and Simkit are efficient tools in producing versatile models. 
Parameters can be changed easily to evaluate the systems. Modification can be made to 
the DES models to simulation other weapons platforms, specifically the JSF. However, 
before using the models, their validity needs to be determined. As a partial effort towards 
this, a stochastic model was formulated and compared with the simulation. This 


stochastic model is described in the following chapter. 
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IV. STOCHASTIC MODELS 


Analytical models produce closed form expressions for MOEs. The use of an 
analytical model in the thesis is for comparison to the DES models. The results from the 
stochastic models are used to partially validate the DES models. The traditional and ALS 
systems can be modeled using stochastic models. The intent is not to assess the entire 
logistic processes using stochastic models. Aircraft engine time between failures and 
repair times are used to approximate the long run average rate of failure and the 
operational availability of the engine. The long run average rate of failure is the number 
of failures per flight hour (FPF). The operational availability is the long run proportion of 
time the aircraft is up. The I level and the D level are not captured in the stochastic 
models. The output of the DES model and the stochastic model results are compared 


under the same model assumptions. 


A. ASSUMPTIONS AND DES MODEL 

Each modular lifetime distribution is assumed to be exponentially distributed with 
a mean of 1,000 hrs. Repair time of an aircraft is constant. If a failure occurs, repair time 
is three hours. If prognostic detects the impending failure, repair time is two hours. The 
additional hour of repair is added for a failure because trouble shooting is required to 


isolate the problem. A repair returns the aircraft to as good as new. 


The parameters listed above are used in the DES models are set to the above 
values. The stochastic models assume a system (aircraft) has two modes: in-use or not- 
in-use. System modes are set to in-use until a failure occurs. During repairs, systems are 
in-use. After the repairs are complete, systems return to the in-use mode. The DES 
models are modified to represent this assumption for comparison with the stochastic 
analytical models. For comparison runs only, aircraft are scheduled for a single long 
flight. A two-year flight duration is used as an arbitrarily large value. The flight is only 
interrupted if a failure occurs and or an impending failure is detected. Immediately upon 


repair the aircraft continues its mission. Each simulation run is replicated 100 times. 
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Each DES model result is the mean of the 100 replications. A discussion of sample size 


selection is included in section D of Chapter V. 


B. RENEWAL PROCESS 
1. Module Failures Process as a Renewal Process 
First, a renewal process is defined, and then counting the number of module 


failures is shown to satisfy the definition. 


If the sequence of times between events { X,,X,...} 1s independent and 

identically distributed, then the counting process { V(t), t=>0} is said to 

be a renewal process (Ross, 2003). 

The nonnegative variables denote the time between the (n-1)st and nth events. An 
event is a renewal. Consider a module, time between module failures is nonnegative and 
independent and identically distributed. Therefore, counting the times a module fails is a 
renewal process. In addition, allowing the module failures to be exponentially distributed 
makes the process a Poisson process. The Poisson process has unique properties which 


1 


make formulation easier. The rate of a Poisson process is 4 = : 
mean 





2. Aircraft Failures as a Poisson Process 

An F-18/A aircraft has two engines. Each engine has twelve modules. The 
aircraft engine failure process is the sum of the twelve independent module failure 
Poisson processes. The sum of independent Poisson processes is a Poisson process with 


arate equal to the sum of the rates of the independent processes (Ross, 2003). 


As a result: 1 =A, +4, +..44 = aa =0.012. Where: 4 is the rate of failure of an 


aircraft, 2 is the rate of the nth module and 4, =A, =..=A, 





3. Long Run Average Failure Rate 
The average rate of a renewal process converges to the reciprocal of the expected 


value of time between two consecutive events as time approaches infinity. 


NO > ! as too (Ross, 2003) 
t E[X] 





32 


For the Poisson process, inter arrival times are exponentially distributed. The 
expected value of the exponential process is the reciprocal of its rate2. Therefore, the 


long average rate of a Poisson process is 1/(1/A) =. 


4. The Aircraft Repair Process as a Renewal Process 

Formulation of the failures is already developed in part three of this section. 
However the aircraft is also subject to repair. Repair times are assumed constant and are 
independent and identically distributed. Let R(t) be the number of repair completions 
and let c be the repair time. R(t)is a renewal process. For the model the expected time 
between repair completions is 2 +c. 

a. The Renewal Reward Process and Long Run Average Availability 

For a renewal process, suppose at each interarrival time X a reward is received. 


Let A be the reward received at each renewal. Let Z(t) represent the total reward 


earned by timer. Z(t)is called a renewal reward process. 


The tim a (Ross, 2003). In other words, the long run average rate of 
renewal reward process is the expected amount of reward per cycle divided by the length 
of the cycle. As a result the long run average rate of availability is the expected time a 


device is up in a cycle divided by the expected length of the cycle (Jacobs, 2006). For our 


model this translates to: long run average rate of availability is ; 
+C 


C. VALIDATING THE TRADITIONAL SYSTEM 

The traditional system enforces an age replacement policy. Modules are replaced 
upon failure or upon reaching a predetermined age. 

iL Age Replacement Policy and Long Run Average Failure Rate 

For a Poisson process, the long run average failure rate for the age replacement 
policy is the same as the long run average failure rate as replacing components only upon 


failure. A proof of this statement is provided in Appendix F. 


The long average rate of failure is: 2 = Th =0.012/ flighthour . 
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The traditional DES model yields:0.01094  standarderror = 0.0003. 


|0.012—0.01094| _ 
0.012 


The relative difference is: 


2. Long Run Portion of Time an Aircraft is Up 


1 1000 
2 


The expected time between failures is ae The expected repair time is three 


hours. The expected time between repair completions is: a +3. 





1000 K 
2 


The long run availability rate is: ; a 


+ 3) =0.9653. 
2 


The traditional DES model yields:0.9682  standarderror = .001. 


[0.9653 - 0.9682] _ 
0.9653 


The relative difference is 0.3%. 





D. VALIDATING THE ALS 

Up to this point, the traditional system has been discussed. Changes to the 
traditional stochastic model are made to account for the detection capabilities of the ALS. 
If prognostics work the impending failure is detected and the module is replaced before 


failure. If the prognostics do not work the module fails. 


The counting of module failures occur is a Poisson process with rate 2. The ALS 
model is as follows. Each failure is associated with an independent trial. If the 
prognostics work the trial is a success. Let p be the prognostic accuracy, then the 
probability of success isp. The probability of a failure, the prognostics did not work, 
isl—p. Let L(t)be the number of failures, then L(t)is a Poisson process with 
rate 2x (1— p) (Jacobs, 2006). 

iL. Long Run Average Failure Rate 


12x (1-0.9) 


Let p=0.9, then the long run average failure rate is = 0.0012. 


The ALS model yields:0.00138 standarderror = .00196. 
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[0.0012 0.00138] _ 
0.0012 


The relative difference is: 


2s Long Run Average Availability Rate 


The expected time between failures is -- ~~ : 


The expected repair time is (3x(1— p))+(2xp). 


The expected time between repair completions is a +(3x(1— p)) + (2x p). 


Let p=0.9, then the long run average failure rate is: 


oS 1 +(3x(1— p))+(2x p)) = 0.997486. 


The ALS DES model yields: 0.9956  standarderror = .00057. 


|0.997486 —-0.9956| 
0.997486 


The relative difference is: = 0.18%. 


E. CONCLUSIONS 

The stochastic models are simple and can be used to provide a solution quickly. 
However, stochastic models are low resolution. Stochastic models capture the general 
overall behavior of the logistic system and are used to checked higher resolution 
simulation models. The simulation models pass face validation. Although this does not 
constitute a complete validation of the DES models, matching the analytical results gives 
confidence to the DES models’ correctness. The DES models are required to study the 
entire engine process in detail. The next step is to simulate the traditional and logistic 


system using the DES models. 


aD 
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V. RESULTS AND DISCUSSION 


This chapter discusses data generation, MOEs and design factors. A Latin 
Hypercube design is used to derive maximal information in a reasonable number of runs. 
Simulation input and output for both the traditional system and the ALS is also 


addressed. 


A. DESIGN OF SIMULATION EXPERIMENTS 

1. Measures of Effectiveness 

Within the DOD, the most common MOE for aircraft readiness is operational 
availability. Operational availability is analyzed as a response variable. However, 
operational availability can be misleading. Operational availability does not indicate how 
often aircraft are used or how often they fail. If a squadron with flyable aircraft did not 
fly for a year, the operation availability for the year is 100%. In this instance, no 
conclusions can be made about the logistic system in place. Other indictors also should 


to be considered in order to get a more accurate picture of the logistic system. 


Other MOEs or response variables considered are: MMHF and FPM. FPM is not 
the same as FPH, which was used in the previous chapter. Both of these response 
variables are directly proportional to cost and provide insight to the effectiveness of 
allocating parts and personnel. 

iz Design Factors 

Design factors are the independent variables considered as predictors for the 
MOEs. The traditional system has four design factors: module reliability, build window, 
depot-turn-around-time and inventory levels. The ALS system has six design factors: 
module reliability, detection lead time, prognostic accuracy, false positive rates, depot 
turn around times and inventory levels. The design factors are analyzed to determine 


their influence on each of the MOEs. 


The baselines for both logistic systems used to compare all results are listed in 
Table 6 of Appendix A. The design factors and the values they can take are summarized 


in Table 2 below. Module reliability is increased by increasing the mean of each 


aT 


distribution. Inventory levels for each module are simultaneously increased with values 
from the table. Detection lead time is the amount of life remaining on a module once an 
impending failure is detected. False positive rates are exponentially distributed with the 


mean values indicated in Table 2. 








Design Factor Values 
Module reliability (hrs) 0,100,200,300,400,500,600,00,800,900, 1000 
Build Window (hrs) 50, 100, 150, 200,250,300,350,400,450,500 





Depot Turn Around (days) 5,10,15,20,25,30,35,40 
Detection Lead Time (hrs) 15,20,25,30,35,40 




















Prognostic Accuracy 0.90,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99,1.0 
False Positive Means (hrs) 450, 500,550,600,650,700 
Inventory (modules) 0,1,2,3,4,5,6,7,8,9, 10 


Table 2. Design Factors and Ranges 


A common technique in experiment design is to run the model using a baseline, 
additional runs are made changing one design factor/predictor variable at a time. 
Predictor variables with the greatest impact are then selected as significant factors. This 
approach is incomplete because: significant combinations of design factors may be 
overlooked. For the full factorial design, adding more design factor levels results in 
numerous required simulation runs. Fractional factorial designs reduce the number of 
runs, but introduce confounding of interactions between design factors. Key interactions 
may be masked or confounded by the main effects (first order effects). 

3. Latin Hypercubes 

A full factorial design considers factor interactions. Such a design for all 
combinations of the four predictor variables for the traditional model would require 9,680 
design points. A full factorial design of all combinations of the six predictor variables for 
the ALS model would require 383,328 design points. Fortunately, other methods can be 


used to construct valid confidence estimates for the MOEs with fewer design points. 


Latin Hypercubes are a very good all-purpose design, particularly when factors 
are quantitative because of: efficiency, space-filling, design flexibility, and analysis 
flexibility (Sanchez, 2006). The nearly-orthogonal Latin Hypercube (NOLH) design is 
used for this experiment. A spreadsheet written by Professor Susan Sanchez, Naval 


Postgraduate School, was used to generate seventeen design points for each system. 
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B. SIMULATION INPUT AND OUPUT 

The model input and outputs are summarized in Table 3 and Table 4. Table 3 
refers to the traditional model. Table 4 refers to the ALS model. The inputs are the 
design factors. The outputs are the MOEs. Each row is a design point (simulation run) 
with its corresponding output. Each simulation run is replicated 100 times. The 
corresponding output is the mean of the 100 replications. The following section is a 


discussion of the sample size detection. 


Traditional Model 













































































Input Output 

Reliability |Depot Inventory |Build (Availability |FPM MMHF 

(hrs) (days) (modules) (hrs) 
400 960 8 200 0.9962) 0.0176 0.3702 
200 360 ) 300 0.996} 0.0207 0.3956 
200 480 1 150 0.9956] 0.0206 0.4202 
300 600 3 500 0.9965) 0.0191 0.3664 
800 960 4 100 0.9955) 0.0133 0.3504 
1000 360 4 400 0.9964] 0.0123 0.2826 
700 240 10 200 0.9967] 0.0143 0.3288 
600 840 8 450 0.997} 0.0155 0.3163 
500 600 5 300 0.9966] 0.0165 0.3405 
700 120 2 350 0.997} 0.0145 0.3097 
900 720 1 250 0.9953] 0.0128 0.3032 
900 600 ) 400 0.9965) 0.013 0.2888 
800 480 7 500 0.9971] 0.0138 0.2936 
300 120 6 450 0.9964) 0.0192 0.3676 
100 720 6 150 0.9954) 0.0223 0.443 
400 840 0 350 0.9965) 0.0178 0.3577 
500 240 3 100 0.996} 0.0162 0.3818 

Depot = depot turn around time (days) 





Table 3. = Traditional System Input and Output 
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ALS 



















































































Input Output 
Reliability [Depot Inventory |Prognostics False Lead Availability|FPM MMHF 
(hrs) (days) (module) (fraction) (hrs) (hrs) 
300 40 8 0.94 500 40 0.997} 0.0011 0.266 
100 15 ) 0.96 450 25 0.9966} 0.0009 0.306 
100 20 1 0.93 600 35 0.9966} 0.0017; 0.3108 
200 25 3 1 600 20 0.9969 O| 0.2753 
800 40 4 0.91 550 15 0.9978} 0.0012) 0.1976 
1000 15 4 0.98 450 35 0.998} 0.0002 0.177 
600 10 10 0.93 650 25 0.9975} 0.0011] 0.2183 
600 35 8 0.99 650 30 0.9976} 0.0002) 0.2136 
500 25 5 0.95 600 30 0.9974, 0.0008) 0.2296 
700 5 2 0.96 650 15 0.9977| 0.0005} 0.2044 
900 30 1 0.94 700 30 0.9979} 0.0007; 0.1867 
900 25 9 0.98 550 20 0.9979} 0.0002) 0.1853 
800 20 7 0.9 550 35 0.9978} 0.0013) 0.1995 
300 5 6 0.99 600 40 0.9971} 0.0002) 0.2587 
0 30 6 0.92 700 20 0.9995} 0.0016) 0.3057 
400 35 0 0.97 500 30 0.9972} 0.0005) 0.2439 
400 10 3 0.91 500 25 0.9973} 0.0016) 0.2481 
F= False positive rate 
Depot = Depot turn around time 
Lead = Detection Lead Time 
Table 4. ALS Model Input and Output 


C. 


POWER ANALYSIS AND SAMPLE SIZE SELECTION 


Power analysis is used to determine sample size (number of replications required) 





based on a f (probability of a type II error) of 0.1. Power is 1- @ which is 0.9. The 
(probability of a type I error), a , is set to 0.05. The model being evaluated is: Response 
variable = (true mean) + (factor effect) + (error). If the null hypothesis is true, then the 
factor effect is zero. The sample size required to detect a deviation in the true mean due 
to a non-zero factor effect is determined. To detect a smaller effect a larger sample size 


is required. 


JMP statistical software is used to calculate the power curve. An estimate of 
standard deviation is needed to perform the calculation. Standard deviation is estimated 
by repeating each design point 100 times and calculating sigma-hat. Figure 11 is 


generated to determine the sample size for operational availability in the traditional 
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logistic model. The graph is a 0.9 power curve. For deviations of 0.005, 45 replications 


are needed to detect that deviation with a probability of 0.90. 
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One Mean 
Alpha 
0.050 


Error Std Dev Power 
0.01 0.9 


o 
iN 
wn 
2 

[om 

= 

o 
wn 





.000 .001 .002 .003 .004 .005 .006 .007 .008 .009 .010 
Difference 





Figure 11. 0.9 Power Curve 


A 0.90 power curve is computed for each MOE in each model by using the same 
procedure. The results are summarized in table 5. All effects detected are smaller than 
0.008. The largest n required is: 70. With 100 replications smaller deviations can be 


detected with a power of 0.90. 




















4] 








Traditional Logistic System ALS 
sample size deviation sample size deviation 
Operational 45 0.005 60 0.005 
FPF 45 0.0015 70 0.0075 
FPM 45 0.0020 60 0.005 
MMHF 60 0.0020 60 0.006 
Table 5. | Power Analysis results. 





D. ANALYSIS OF THE SIMULATION OUTPUT 
Each simulation produces three output MOEs. Analysis of the MOEs for both the 


ALS and traditional logistic system served to compare both models. 


The intent of the simulation is to study the effects of the design factors on each of 
the MOEs. To gain insight into the relationship of the variables, linear regression models 
are fitted to the generated data. The regression models are a result of using stepwise 
regression and residual analysis. The stepwise algorithm adds and drops terms by using 
the Akaike’s Information Criterion. The P-value is the probability the effect of a variable 
equals zero. Variable with P-value greater then 0.001 are not considered. The smaller the 
P-value the more evidence there is against the null hypothesis: the predictor variable has 
no effect. If diagnostic plots do not support the modeling assumptions of normal errors 
and homoscedasticity, mathematical transformations are applied until the assumptions are 
satisfied. Transformations include: logarithmic, reciprocal, arcsine and square root. 
Additionally, if the relationship between the response variable and the predictor variables 


is not linear, polynomials of degree two are used for the regression. 


JMP statistical software is used to perform multiple regression and to generate 
graphs to summarize the analysis. For each MOE, linear regression models are produced 
to describe its relationship with the input variables of each system. Scatter plot matrices 
are used to check the pairwise relationship between all variables. When two or more 
predictor variables are found to be significant a correlation matrix is generated to verify 
they are independent. Contour plots are used to compare the value of predictor variable 
pairs for an MOE. The contour values are plotted from the results of the 17 design 
points. The contour plot is not a feasible region and it may or may not contain the 


optimum point. 


The predictive capability of each model is verified by exploring inputs not already 
used. A specific combination of design factors yields the optimal value for each MOE. 
An optimal solution is sought for each MOE: maximum operational availability, 
minimum FPM and maximum MMHEF. The optimal solution is found for each linear 
model. The DES model is then run with the optimal solution. The output generated by 


inputs for the DES model and linear model are compared. Outputs close in value indicate 
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an adequate linear model. It is important to note the linear models are valid for specific 


predictor variable ranges. Values outside the define ranges from table 2 are not explored. 


A linear model with predictive capability can provide as much insight as the more 
computationally intensive DES model. Additionally, a linear model can be put in an 
Excel spreadsheet which is easier and quicker to use. For FEM and MMHF, linear models 
are be used to compare the traditional system and the ALS. 

iL: Traditional Repair System 

Data generated by the DES traditional repair system model resulted in good 
predictive models for FPM and MHPF, but not for operational availability. 

a. Operational Availability 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, build window, depot turn around time, and 
inventory, and the response variable operational availability. Figure 12 is the JMP 
regression report. The parameter estimates table of the report gives the estimated 
coefficients, their standard errors and the corresponding t-statistics to test the partial 
effect of each variable. The R’ for this least square fit is 0.503, indicating the regression 
model does not adequately explain the variation in operation availability. Build windows 
is the only predictor variable kept in the model. Inventory and depot turn around time 
were anticipated to be significant in terms of operational availability. Not being selected 
for the model does not mean they are not important. For the predictor variable ranges 
from table 3, inventory and depot turn around time do not limit operational availability. 
For the traditional logistic system, I-level inventory equivalent to that from table 6 with a 
guaranteed depot turn around time of 40 days is all that is required. In other words, once 
at this level increasing the I-level inventory or decreasing the depot turn around time has 
a minimal effect on operational availability. At this level, the main driver for operational 


availability is build windows. 
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buildwindows 
Summary of Fit 
RSquare 0.503204 
RSquare Adj 0.470084 
Root Mean Square Error 0.000412 
Mean of Response 0.996276 
Obser ations (or Sum Wgts) 17 
Analysis of Variance } 
Source DF Sum of Squares Mean Square F Ratio 
Model 1 0.00000258 0.0000026 15.1935 
Error 15 0.00000255 1.6992e-7 Prob >F 
C. Total 16 0.00000513 0.0014 
| Lack Of Fit 
Source DF Sum of Squares Mean Square F Ratio 
Lack Of Fit 7 0.00000161 2.2984e-7 1.9561 
Pure Error 8 0.00000094 1.175e-7 Prob > F 
Total Error 15 0.00000255 0.1836 
Max RSq 
0.8168 
Parameter Estimates 
Term Estimate Std Error t Ratio Prob>|t| 
Intercept 0.9953838 0.00025 3983.5 <.0001 
buildwindows 0.0000029 7.559e-7 3.90 0.0014 
_Effect Tests 
Source Nparm DF Sum of Squares F Ratic Prob>F 
buildwindows 1 1 0.00000258 15.1935 0.0014 


Figure 12. JMP Regression Report for Traditional System Operational Availability 


b. Failures Per Flight Mission 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, build window, depot turn around time and 
inventory, and the response variable FPM. Figure 13 is the JMP regression report. R’ for 
this least square fit is 0.998, indicating the regression does a good job in accounting for 
the variability in FPM. The parameter estimates table of the report gives the estimated 


coefficients, their standard errors and the corresponding t-statistics to test the partial 
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effect of each variable. Module reliability is the only predictor variable selected for the 
linear model. Build windows is the next significant variable, but not significant enough 


to be included in the model. 
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RSquare 0.998113 
RSquare Adj 0.997844 
Root Mean Square Error 0.000145 
Mean of Response 0.016441 
Observ ations (or Sum Wgts) 17 
Analysis of Variance 
Source DF Sum of Squares Mean Square F Ratio 
Model 2 0.00015573 0.000078 3703.262 
Error 14 0.00000029 2.103e-8 Prob >F 
C. Total 16 0.00015602 <.0001 
Lack Of Fit } 
Source DF Sum of Squares Mean Square _ F Ratio 
Lack Of Fit 7 5.43588e-8 7.7655e-9 0.2265 
Pure Error 7 0.00000024 3.4286e-8 Prob > F 
Total Error 14 2.94359e-7 0.9657 
Max RSq 
0.9985 
Parameter Estimates 
Term Estimate Std Error t Ratio Prob>|t 
Intercept 0.0220116 0.000088 250.42 <.0001 
Module Reliability (Flight hrs) -0.000011 1.302e-7 -85.51 <.0001 
(Module Reliability (Flight hrs)-547.059)*(Module Reliability (Flight hrs)-547.059)7.1084e-9 5.52e-10 12.87 <.0001 
Effect Tests 
Source Nparm DF Sum of Squares F Ratic Prob>F 
Module Reliability (Flight hrs) 1 1 0.00015375 7312.29 <.0001 


Module Reliability (Flight hrs)*Module Reliability (Flight hrs) 1 1 0.00000348 165.5464  <.0001 


Figure 13. JMP Regression Report for Traditional System FPM 
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The scatter plot matrix, Figure 14, indicates there is no collinearity 
between module reliability and build windows. The plot shows a linear relationship 
between module reliability and FPM. This supports the fitted model: 
0.022 — (0.00001 * ModuleReliability) + (0.00000007 * (ModuleReliabilty —547.06)%2) 
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Figure 14. Scatter Plot Matrix Traditional System FPM 
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Figure 15 is a contour plot, comparing module reliability and build 
windows in terms of FPM. The vertical color shading pattern suggests failure rate 
improves with the increase of module reliability and is not affected by build windows. 


This provides further support for the fitted model. 


Contour Plot for FPM 
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Figure 15. Contour Matrix Traditional System FPM 
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Figure 16 is a plot of the FPM linear model of the traditional logistic 
system over the module reliability range. The fitted regression model decreases in FPM 
when module reliability increases. Therefore, the largest value of module reliability in the 
model range produces the smallest FPM. The highest value of the module reliability is 
1000 hrs. Module reliability is set to 1000 hrs. The regression model produces an FPM 
of 0.01234. With module reliability fixed at 1000 hrs, the DES model for the traditional 
repair system produces an FPM of 0.0125 with standard error 0.00004. The difference 


between the regression model and the simulation model is 1.28 %. 
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Figure 16. FPM Linear Regression Model for Traditional Repair System 


c. Maintenance Man Hours Per Flight Hour 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, build window, depot turn around time and 
inventory, and the response variable MMHF. Figure 17 is the JMP regression report that 
includes a correlation matrix.. R* is 0.964, indicating the regression does a good job in 


accounting for the variability in MMHF. The parameter estimates table of the report gives 
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the estimated coefficients, their standard errors and the corresponding t-statistics to test 
the partial effect of each variable. Module reliability and build windows are the predictor 
variables selected for the linear model. Module reliability and build windows have 


approximately equal weight on the expected value of MMHF. 
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MMHF Predicted P<.0001 RSq=0.96 
RMSE=0.0093 
| Summary of Fit } 
RSquare 0.964953 
RSquare Adj 0.959946 
Root Mean Square Error 0.009308 
Mean of Response 0.348024 
Obser ations (or Sum Wots) 17 
| Analysis of Variance } 
Source DF Sum of Squares Mean Square _ F Ratio 
Model 2 0.03339347 0.016697 192.7328 
Error 14 0.00121284 0.000087 Prob >F 
C. Total 16 0.03460631 <.0001 
Param eter Estimates ] 
Term Estimate Std Error t Ratio Prob>|t| 
Intercept 0.4632566 0.006733 68.81 <.0001 
Module Reliability (Flight hrs) -0.000141 0.000008 -16.59 <.0001 
buildwindows -0.000126 0.000017 -7.24 <.0001 
Effect Tests 
Source Nparm DF Sum of Squares F Ratic Prob >F 
Module Reliability (Flight hrs) 1 1 0.02384449 275.2404 <.0001 
buildwindows 1 1 0.00454368 52.4484 <.0001 
Correlation of Estimates 
Cor 
InterceptModule Reliability (Flight hrb)jildwindows 
Intercept 1.0000 -0.5456 -0.6536 
Module Reliability (Flight hrs) -0.5456 1.0000 -0.1853 
buildwindows -0.6536 -0.1853 1.0000 


Figure 17. Traditional MMHF Regression Model 
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The scatter plot matrix, Figure 18, indicates there is no collinearity 
between module reliability and build windows. The plot shows a moderate linear 
relationship between build windows and MMHF. However, the plot shows a slightly 
stronger linear relationship between module reliability and MMHF. This supports the 


fitted regression model: 0.46 — (0.00014 * reliabilty) — (0.000125 * buildwindow) . 
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Figure 18. Scatter Plot Matrix Traditional System MMHF 
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Figure 19 is a contour plot, comparing module reliability and build 
windows in terms of MMHF. The color shading forms a forty-five degree pattern 
suggesting module reliability and build windows are nearly equivalent predictors of 
MMHF._ This provides further evidence in support of the fitted regression model. 
Additionally, build window has no effect on MMHF when module reliability is 800 hrs 


or greater. 
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Figure 19. Contour Plot Traditional System MMHF 


Figure 20 is a plot of the MMHEF linear model of the traditional repair 
system over the build window range with no improvement in module reliability range. 
Figure 21 is a plot of the MMHE linear model of the traditional repair system over the 
module reliability range and a constant build window of 500 hours. The fitted regression 
model predicts a decrease in MMHF when module reliability increases or when build 
window increases. Linear programming is used to optimize the MMHF regression model 
with the predictor variable ranges as constraints. Setting both module reliability and 
build window to the highest value in their respective range yield the optimal (smallest) 
MMHF. Module reliability is set to 1000 hrs and build window is set 500 hours. The 
regression model produces an MMHF of 0.2954. The DES model for the traditional 
repair system produces an MMHF of 0.2755 with standard error 0.000047. The 


difference between the regression model and the simulation is 7.2%. 
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Figure 20. MMHF Regression Model for Traditional Repair System 
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Figure 21. MMHF Regression Model for Traditional Repair System 
52 








2. Analysis of ALS Output 

Data generated by the DES ALS model resulted in good predictive models for the 

FPM and MMHEF but not for operational availability. 

a. Operational Availability 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, prognostics accuracy, false positives, depot 
turn around time, inventory and detection lead time, and the response variable operational 
availability. Figure 22 is the JMP regression report. The parameter estimates table of the 
report gives the estimated coefficients, their standard errors and the corresponding t- 
statistics to test the partial effect of each variable. The R’* value is 0.14, indicating the 
linear regression model does not explain the variation in operation availability. This 
suggests the selected multiple regression model predicts poorly. Module reliability and 
prognostic accuracy are the only predictor variables kept in the model. Inventory and 
depot turn around time were anticipated to be significant in terms of operational 
availability. Not being selected for the model does not mean they are not important. For 
the specific variable ranges from table 4, inventory and depot turn around time do not 
limit operational availability. For the ALS, I-level inventory equivalent to that from table 
6 with a guaranteed depot turn around time of 40 days is all that is required. In other 
words, at these specific values increasing the I-level inventory or decreasing the depot 
turn around time has a minimal effect on operational availability. At these specific 
values, the main driver for operational availability is prognostic accuracy followed by 


module reliability. 
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| Summary of Fit 

















RSquare 0.142536 
RSquare Adj 0.020041 
Root Mean Square Error 0.00067 
Mean of Response 0.997518 
Obser ations (or Sum Wgts) 17 
Analysis of Variance 

Source DF Sum of Squares Mean Square F Ratio 
Model 2 0.00000104 5.2202e-7 1.1636 
Error 14 0.00000628 4.4862e-7 Prob>F 
C. Total 16 0.00000732 0.3408 

Param eter Estimates } 
Term Estimate Std Error t Ratio Prob>|t| 
Intercept 1.0025573 0.005092 196.88 <.0001 


Module Reliabilty (flight hrs) 5.9124e-7 5.347e-7 1.11 0.2874 
Prognostics Accuracy (%) -0.005616 0.005347 -1.05 0.3113 








Effect Tests } 
Source Nparm DF Sum of Squares  F Ratic Prob>F 
Module Reliabilty (flight hrs) 1 1 5.48618e-7 1.2229 0.2874 
Prognostics Accuracy (%) 1 1 4.95023e-7 1.1034 0.3113 


Figure 22. JMP Regression Output for ALS Operational Availability 


b. Failures Per Mission 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, prognostics accuracy, false positives, depot 
turn around time, inventory and detection lead time, and the response variable FPM. 
Figure 23 is the JMP regression report with correlation matrix attached. The parameter 
estimates table of the report gives the estimated coefficients, their standard errors and the 
corresponding t-statistics to test the partial effect of each variable. R* is 0.98, indicating 
the regression does a good job in accounting for the variability in FPM. Module 


reliability and prognostic accuracy are the two predictor variables selected for the linear 
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model. Prognostic accuracy has the heaviest weight value on the expected value of FPM 


followed by module reliability and the interaction between prognostics accuracy and 


module reliability. 
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001 -002 
FPM Predicted P<.0001 RSq=0.98 
RMSE=8.5e-5 
Summary of Fit 
RSquare 0.981146 
RSquare Adj 0.976796 
Root Mean Square Error 0.000085 
Mean of Response 0.000812 
Observ ations (or Sum Wgts) 17 
Analysis of Variance 
Source DF Sum of Squares Mean Square F Ratio 
Model 3 0.00000486 0.0000016 225.5073 
Error 13 9.34697e-8 7.19e-9 Prob>F 
C. Total 16 0.00000496 <.0001 
Parameter Estimates 
Term Estimate Std Error t Ratio Prob>|t 
Intercept 0.0167315 0.000646 25.91 <.0001 
Module Reliabilty (flight hrs) -5.982e-7  6.771e-8 -8.83 <.0001 
Prognostics Accuracy (%) -0.016429 0.000678 -24.23 <.0001 
(Module Reliabilty (flight hrs)-505.882)*(Prognostics Accuracy (%)-0.95059) 0.00001 0.000002 4.40 0.0007 
Effect Tests 
Source Nparm DF Sum of Squares F Ratic Prob >F 
Module Reliabilty (flight hrs) 1 1 5.61225e-7 78.0566 <.0001 
Prognostics Accuracy (%) 1 1 0.00000422 587.2160 <.0001 
1.3944e-7 19.3937 0.0007 


Module Reliabilty (flight hrs)*Prognostics Accuracy (%) 1 1 


Corr 

Intercept R P(R-505. 882)*(P-0.95059 
Intercept 1.0000 -0.0518 -0.9981 0.0557 
R -0.0518 1.0000 -0.0012 0.0283 
P -0.9981 -0.0012 1.0000 -0.0573 
(R-505.882)*(P-0.95059) 0.0557 0.0283 -0.0573 1.0000 


R = module reliability, P= prognostic accuracy 


Figure 23. JMP Regression Report for ALS FPM 
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The scatter plot matrix, Figure 24, indicates there is no collinearity 
between module reliability and prognostic accuracy. The plot shows a moderate linear 
relationship between module reliability and FPM. However, the plot also shows a 


stronger linear relationship between prognostic accuracy and FPM. This supports the 


fitted regression model: 


0.167 —(5.9e - 7 * reliability) - (0.016 * prognostics) + (0.0001 * (reliability - 505.9) * (prognostics - 0.95) 
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Figure 24. Scatter Plot Matrix ALS FPM 
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Figure 25, is a contour plot, comparing module reliability and prognostic 
accuracy in terms of FPM. Note the color pattern tends to be horizontal and slightly 
tilted, indicating prognostic accuracy is more important. The tilt in the contour plot 
indicates including an interaction between module reliability and prognostic accuracy 
helps predicting FPM. The interaction term is included in the regression. This provides 


further evidence in support of the fitted regression model. 
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Figure 25. Contour Plot ALS FPM 


Figure 26 is a plot of the MMHEF linear model of the traditional repair 
system over the build window range with no improvement in module reliability range. 
The fitted regression model predicts a decrease in FPM when prognostic accuracy 
increases or when module reliability increases. Linear programming is used to optimize 
the FPM regression model with the predictor variable ranges as constraints. Setting both 
prognostic accuracy module reliability to the highest value in their respective range yield 
the optimal (smallest) FPM. Prognostic accuracy is set to 1.0 and module reliability is set 
1000 hrs. The regression model produces an FPM of 0.00005. The DES model for the 
ALS produces an FPM of 0.0 with standard error 0.0. The relative difference between the 


regression model and the simulation models is irrelevant as both essentially equal zero. 
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Figure 26. FPM ALS Regression Model for ALS 


c Maintenance Man Hours Per Flight Hour 

Regression is used to fit a linear model to explain the relationship between 
the predictor variables: module reliability, prognostics accuracy, false positives, depot 
turn around time, inventory and detection lead time, and the response variable MMHF. 
Figure 27 is the JMP regression report. The parameter estimates table of the report gives 
the estimated coefficients, their standard errors and the corresponding t-statistics to test 
the partial effect of each variable. R° is 0.984, indicating the regression does a good job 
in accounting for the variability in MMHF. Module reliability is the only predictor 
variable chosen for the linear model. False positive is the next significant variable, but 


not significant enough to be included in the model. 
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RSquare 0.984845 
RSquare Adj 0.98268 
Root Mean Square Error 0.005854 
Mean of Response 0.236853 
Obser ations (or Sum Wgts) 17 
Analysis of Variance 
Source DF Sum of Squares Mean Square F Ratio 
Model 2 0.03117302 0.015587 454.8869 
Error 14 0.00047970 0.000034 Prob >F 
C. Total 16 0.03165272 <.0001 
| Lack Of Fit } 
Source DF Sum of Squares Mean Square _ F Ratio 
Lack Of Fit 8 0.00041889 0.000052 5.1659 
Pure Error 6 0.00006082 0.000010 Prob > F 
Total Error 14 0.00047970 0.0303 
Max RSq 
0.9981 
( Parameter Estimates 
Term Estimate Std Error t Ratio Prob>|t 
Intercept 0.3013858 0.003221 93.57 <.0001 
Module Reliabilty (flight hrs) -0.000139 0.000005 -29.85 <.0001 
(Module Reliabilty (flight hrs)-505.882)* (Module Reliabilty (flight hrs)-505.882)6.5345e-8  1.75e-8 3.73 0.0022 
| Effect Tests 
Source Nparm DF Sum of Squares F Ratic Prob>F 
Module Reliabilty (flight hrs) 1 1 0.03052350 890.8178 <.0001 


Module Reliabilty (flight hrs)*Module Reliabilty (flight hrs) 1 1 0.00047757 13.9378 0.0022 


Figure 27. JMP Regression Report for ALS MMHF 
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The scatter plot matrix, Figure 28, indicates there is no collinearity 
between module reliability and false positives. The plot shows a linear relationship 
between module reliability and MMHF. This supports the findings of the regression 


model: 0.3- (0.00014 * reliability) - (6.5e -8 * (reliability -505.88)*2) . 
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Figure 28. Scatter Plot Matrix ALS MMHF 
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Figure 29, is a contour plot, comparing module reliability and false 
positives in terms of MMHF. The vertical coloring pattern suggests failure rate improves 
with the increase of module reliability and is not affected by false positives. This 


provides further evidence in support of the fitted regression model. 
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Figure 29. Contour Plot ALS MMHF 


Figure 30 is a plot of the MMHEF linear model of the ALS system over the 
module reliability range. The fitted regression model predicts a decrease in MMHF when 
module reliability increases. Therefore, the largest value of module reliability in the range 
is used in the regression model to calculate the smallest MMHF. The highest value of the 
module reliability is 1000 hrs. Module reliability is set to 1000 hrs. The regression model 
produces an MMHF of 0.17785. The DES model for the ALS produces an MMHF of 
0.1590 with standard error 0.0016. The relative difference between the regression model 


and the simulation model is: 11.8%. 
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Figure 30. MMHF Regression Model for ALS 


E. COMPARING THE TRADITIONAL LOGISTIC SYSTEM TO ALS 

For FPM and MMHEFE linear models can be used to compare the traditional system 
and the ALS. For operational availability a screening technique is used to compare the 
traditional system and the ALS. 

A, Comparison in Terms of Operational Availability. 

The data initially generated consisted of 17 design points. For this data, 
operational mean for the traditional system is 0.996726 with a standard deviation of 
0.000566 and for the ALS is 0.997518 with a standard deviation of 0.0006777. The ALS 
has the larger mean, but it also has the largest standard deviation. The difference 


between the two systems is not clear. 


Rinott’s two-stage screening method is used to select which logistic system is the 
best. The method is fully explained in (Chen & Kelton, 2000). To use this method it is 


assumed the data sets are normally distributed. To satisfy the assumption each DES 
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models was run an additional thirty times. The thirty design points were selected using 
NOLH. The design points and the corresponding output are listed in Appendix G. In 
total there are 46 observations for each system. Each data set may be considered 
normally distributed by applying the general rule of thumb: if a data set is greater than 
thirty the Central Limit Theorem (CLT) can be used (Devore, 2004, 240). The CLT state 


large independent random samples (data sets) are approximately normally distributed. 


A spreadsheet written by Professor Susan Sanchez, Naval Postgraduate School is 
used to implement Rinott’s two-stage screening method. The ALS is chosen as the best 
system after the first stage. The second stage is not necessary. 

Zz Comparison in Terms of FPM 

For FPM, the regression indicates module reliability is the most important factor 
for the traditional system and prognostic accuracy is the most important factor for the 
ALS. Constraints may preclude investing in both module reliability and prognostic 


accuracy. 


The optimal (smallest) value for the FPM linear model of the traditional repair 
system is 0.012, when module reliability is 1000 hours. The FPM linear model for the 
ALS includes prognostic accuracy and module reliability. Since only one variable can 
change, module reliability is set to a constant zero. This means module reliability is not 
improved. The optimal (smallest) value is 0.00005 when prognostic accuracy is 1.0. The 
worst (largest) value is 0.001 when prognostic is 0.9. For the parameter ranges, the 
prognostic worst case for the ALS yields a lower FPM then the best case for the 
traditional system. In this case, investing in prognostics is clearly better than investing in 


module reliability. 


In the case when one option does not dominate, graphs like Figure 31 can be used 
to find the turning point. The turning point is the values for which one option is better or 
worse than the other. To construct Figure 31, first set the two regression models equal to 
each other and solve for prognostic accuracy. The result is an equation for prognostic 
accuracy in terms of module reliability. This is the equilibrium equation. The pair of 
values of prognostic accuracy and module reliability that satisfy the equation yield the 


same FPM: in terms of FPM they are equivalent. Thus, investing in one or the other is 
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the same. Select values above and below the equilibrium line, to determine which option 
is best. Note the graph of Figure 31 has prognostic accuracy between values of 0.15 and 
0.45. This coincides with the conclusion, but the model prognostic accuracy range is 0.9 
to 1.0. Therefore, for this case Figure 31 should not be used for further analysis. For the 
entire parameter ranges the ALS performs better than the traditional repair system. When 
deciding between investing in module reliability in the traditional repair system or 


prognostic accuracy it is best to invest in prognostics and switch to ALS. 
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Figure 31. FPM Regression Models, Traditional Repair System Versus ALS 


3. Comparison in Terms of MMHF 

For MMHF, the regression indicates build window and module reliability are the 
most important factors for the traditional system and module reliability is the most 
important factor for the ALS. Two questions are answered: (1) can increasing only build 
windows for the traditional logistic yield similar of better results then the ALS, (2) with 
build windows set at the optimal value (500 hrs), is it better to invest in module reliability 


in the traditional logistics system or the ALS. 
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First a comparison is made between the linear model of the traditional logistic 
system over the build window range with no improvement in module reliability range and 
the ALS model over the module reliability range. Figure 32 is a plot of the equilibrium 
equation. When no improvement in the ALS module reliability is made the two systems 
are equivalent when build windows is increased to over 1000 hours. The build window 
range is 50 to 500 hours. Therefore Figure 32 should not be used for further analysis. 
The regression models show that within the parameter ranges of the simulation increasing 
only build windows in the traditional system will yield a higher (worse) MMHF than the 
ALS. 
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Figure 32. MMHEF Regression Models, Traditional Repair System Versus ALS 


Next, a comparison is made between the linear model of the traditional logistic 
system over the module reliability range and build window range set to 500 hours and the 


ALS model over the module reliability range. Figure 33 is a plot of the equilibrium 
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equation. When no improvement in the ALS module reliability is made the two systems 
are equivalent when module reliability is increased by over 500 hours for the traditional 
system. Eventually as displayed in Figure 33 for the systems to be equivalent module 
reliability for the traditional system is increased by over 100 hours, which exceed the 
module reliability range. The regression models show that the deciding between 
investing in module reliability in the traditional logistics system or in the ALS, it is best 


to invest in the ALS. 
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Figure 33. MMHF Regression Models, Traditional Repair System Versus ALS 


F. CONCLUSIONS 

This chapter outlined the results of the thesis. Design points to drive the 
simulations were carefully selected to extract as much information as possible from a 
minimum number of runs. For both systems, regression led to good predictive models for 


FPM and MMHEF, but not operational availability. Additional runs were made to 
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implement Rinott’s screening method to compare the operational availability of both 
systems. For each of the MOEs, the ALS is the better system. The ALS dominates in 
terms of FPM and MMHF. However, large gains in operational availability were not 


realized. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The objective of the thesis was to compare the traditional logistics system to the 
ALS and analyze the hypothetical benefits of the ALS. Key components necessary to 
model the system were identified. DES models were developed using Java and Simkit. 
The DES models and the regression models can be used as decision making tools. The 


regression models, only apply within the predictor variable ranges. 


Conclusions for the traditional repair system: 


L. I-level inventory equivalent to that from table 6 with a guaranteed depot 
turn around time of 40 days is all that is required for operational 
availability. At these specific values increasing the I-level inventory or 
decreasing the depot turn around time has a minimal effect on operational 
availability. At these specific values, the main driver for operational 
availability is build window. 


2 Module reliability is most important in terms of FPM. Shrinking build 
windows at the I-level does not improve FPM. Shrinking build windows 
leads to some parts being replaced early and more often at the I-level. 
Replacing parts more often does not decrease FPM. The parts are on a 
replacement schedule at the O-level. With the replacement (high time) 
schedule in place, replacing parts earlier at the I-level does not improve 
FPM. 


se Module reliability and build windows are the two significant factors for 
predicting MMHF. They are equally weighted. 


Conclusions for the ALS: 


Lis I-level inventory equivalent to that from table 6 with a guaranteed depot 
turn around time of 40 days is all that is required for operational 
availability. At these specific values increasing the I-level inventory or 
decreasing the depot turn around time has a minimal effect on operational 
availability. At these specific values, the main driver for operational 
availability is prognostic accuracy followed by module reliability. 


2 Module reliability and prognostic accuracy are the significant factors in 
the FPM model. Module reliability is weighted more heavily then 
prognostic accuracy. 


J Module reliability is the only significant factor in predicting MMHF. 
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For FPM, the regression models show that when deciding between investing in 
module reliability in the traditional repair system or prognostic accuracy it is best to 
invest in prognostics and switch to ALS. In fact, within the parameter ranges of the 
simulation, the worst FPM rate for ALS was better than the best FPM rate for the 


traditional repair system. 


The analysis shows that when operational availability is the criteria used to 
distinguish between the two systems, the ALS is selected as the best. A performance 
based logistic contract which guarantees delivery of components within a certain time 


frame makes the two logistic systems more equivalent. 


For MMHF, the regression models show that when deciding between investing in 
module reliability in the traditional repair system or to invest in module reliability in the 
ALS, it is best to switch to the ALS. The ALS with no improvements is equivalent to 
increasing both build window and module reliability to the maximum value for the 


traditional system. The ALS potential far exceeds that of the traditional logistic system. 


B. RECOMMENDATIONS FOR FURTHER STUDY 
Potential follow-on-studies include researching the effects of: 
e different module replacement schedules (high time). 
e different flight schedules. 


e longer depot turn around times, to gain an understanding of when depot 
turn around time impacts operational availability. 


* different ranges for the predictor variables. 

Further enhancements to the DES models may be applied: 
e Include the depot level in detail. 

e Model actual individual maintainers and equipment. 


e As the JSF ALS is better defined, adjust the DES. 


c: CONCLUSION AND RECOMMENDATION ABOUT THE ALS 
For each MOE, the ALS out performed the traditional logistics system. The ALS 
with prognostic accuracy of at least 0.9 and an exponential mean time between false 


positives of no more than 700 hours dominates the traditional system in terms of FPM 
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and MMHF. Within the scope of the thesis, the traditional system was never found to be 


the better option. The ALS is superior to the traditional repair system. 
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APPENDIX A. DATA VALUES 


The following data were obtained from Lieutenant Commander Schoch’s thesis 


(Schoch, 2003): 


l-level site chosen: Pax River 















































Number of O-levels: 1 
Number of aircraft: 25 
l-level 
O-level Trouble Inspection 
Part High Times (hrs) I R shooting time Time T 
1100,2000,2200, 
4000,5350,7175, 
Fan 7342,7575,10700 7 | 2.4 N/A N/A N/A 
2000,2967,3075, 
3800,4000,4150, 
Compressor 4717,8117 3 | 2.71 N/A N/A N/A 
Combustor 2000 4 | 1.43 N/A N/A N/A 
1203,1542,2200, 
HPT 2417,2483 2 | 2.27 N/A N/A N/A 
2000,4000,4242, 
LPT 16625,19025 1) 1.31 N/A N/A N/A 
Afterburner 2000 2 1.6 N/A N/A N/A 
Engine N/A 8 | 3.3 3.34 4.5 0 
| = Inventory levels 
R = Install / Removal times (hrs) 
T = O-level to I-level transfer time (hrs) 








Table 6. Data Values 
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APPENDIX B. ALS F18 HORNET EVENT GRAPH 
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APPENDIX C. ALS O-LEVEL EVENT GRAPH 
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ALS I LEVEL EVENT GRAPH 


APPENDIX D. 
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APPENDIX E. ALS D-LEVEL EVENT GRAPH 
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APPENDIX F. AGE REPLACEMENT POLICY 


The following is the long run average failure rate model for the age replacement 


policy by Professor Patricia Jacobs, Naval Postgraduate School: 

An age replacement policy class for an item upon its failure or upon it reaching 
age T whichever occurs first: 

Replacement items are as good as new. Let L, be the i” lifetime. Assume 
L.L,,... are independent identically distributed having a distribution function F with 
density f.Let Y,Y,... denote the times between successive failures. Let N,(t) be the 


number of failures to occur during (0,t]. {N,(¢);t 20} is a possibly delayed renewal 


N (t 
counting process with inter-renewal times {Y,}, lim “ ) = ED] with probability 1. 
too y, 





To find E[Y,]: 


The random variable Y, is comprised of a random number of time periods of 
length T (corresponds to replacements not associated with failures) plus a last time 


period for which the distribution is that of a failure conditioned on failure before age T. 





Y,=NT+U 
P{N>k}=[1-F(1)) for k=1,2... 
pupae sae ne 
P{L<T} 
=0 if u<O 
me if 0O<u<T 
F(T) 
=] if u>T 


Result: Let N be a nonnegative random variable. 


E[N]=>np{N =n} =>P{N2n}=>P{N>n} 
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Using the result in this example 


E(NJ=XP{N2n}=S[l- FY Ao 


E{u]= Siu 





LO 4, 
F(T) 


ElY,]= E[NT + U]= E[NIT + E[U]= HaF@) (ua, 
F(T) F(T) 


(Jacobs, 2005, 12-13 ) 
STATEMENT: If the lifetime distribution is exponential then the log run 


average number of failures under age replacement policy is the same as the long run 


average number of failures when the item is replaced upon failure alone. 


Proof: 


ElY,]= E[NT + U]= EA(NIT + E[U]= eae CO) (yt oa 
F(T) F(T) 
Pa ee? 8) (Se 
F(T) (e") 














Te“ -Te“+[l-e“]/A _ 1 


(l-e”) A 





but : = expected time of failure due to age if lifetime exponentially distributed. 


Therefore, the statement is correct. 
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APPENDIX G. ADDITIONAL SIMULATION RUNS 


Traditional Model 


Reliability | Depot Inventory | Build Availability 
(hrs) window 


|(hrs) __[ (days) _| (modules) | (hrs) | 


_ 


a 


os 


_ 


—_ 


—_ 


8 
7 
8 
1 
7 
4 
2 
0 
2 
9 
1 
6 
4 
8 
4 
5 
3 
1 
7 
1 
5 
0 
6 
1 
2 
9 
3 
6 
0 


a 








Depot = depot turn around time 


Table 7. ALS additional runs 
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Depot False Lead 
Reliability | (hrs) : Postives | Time 


(hrs) __ | (days) _| (module) | (fraction) | (hrs) | (hrs) | 
10 9 


0 


100 30 
600 40 
1000 20 
100 30 
800 10 
1000 20 
700 30 
1000 35 1 
500 10 
900 15 
800 30 1 
700 40 
300 35 
200 35 
1000 40 
700 25 
200 35 
300 20 1 
400 20 
800 25 1 
400 40 
600 30 1 
500 25 
400 40 1 
900 15 
300 15 
900 25 
100 20 
500 10 1 


Input 


ODWONnNHDWDOAHANHWARPAWAKRHDH ONON AN -] ON CO 


ALS 


0.94 
0.97 
0.98 
0.93 
0.94 
0.99 
0.99 
0.95 
0.96 
0.91 
0.99 
0.93 
0.91 
0.92 


600 
650 
700 
650 
650 
550 
600 
500 
500 
650 
550 
650 
700 
600 
600 
650 
550 
600 
650 
500 
450 
650 
550 
650 
600 
650 
500 
550 
500 
500 


15 


Availabilit 


0.9995 
0.9966 
0.9976 

0.998 
0.9966 
0.9978 

0.998 
0.9977 

0.998 
0.9974 
0.9979 
0.9978 
0.9977 
0.9971 
0.9968 

0.998 
0.9977 
0.9968 
0.9971 
0.9972 
0.9978 
0.9973 
0.9976 
0.9974 
0.9972 
0.9979 
0.9971 
0.9978 
0.9966 
0.9974 





F= False positive rate 
Depot = Depot turn around time 
Lead = Detection Lead Time 





Table 8. 


ALS additional runs 
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